Adaptive scaling algorithms often use only one color channel of an image to save cost. Algorithms in the YUV color space typically operate on the luma (Y) channel. The processes typically interpolate chroma (UV) channels using polyphase filtering or other non-adaptive scaling algorithms. This can lead to a situation where the borders of objects in the mage can be smooth in the luma channel, but have jaggies in the chroma channel. This results in noticeable degradation of the image when the colors are highly saturated. Similar problems arise in demosaic operations of imaging sensors that use a Bayer filter, or where the ideal color space for interpolation is different than that for determining how to do the adaptation.